Skip to content

PROP-96 - Add Federated ML to Propeller#97

Closed
JeffMboya wants to merge 41 commits intoabsmach:mainfrom
JeffMboya:PROP-96
Closed

PROP-96 - Add Federated ML to Propeller#97
JeffMboya wants to merge 41 commits intoabsmach:mainfrom
JeffMboya:PROP-96

Conversation

@JeffMboya
Copy link
Contributor

@JeffMboya JeffMboya commented Dec 5, 2025

What type of PR is this?

Feature

What does this do?

Adds ML to WASM demo application

Which issue(s) does this PR fix/relate to?

Have you included tests for your changes?

Yes, manually tested

Did you document any new/modified features?

Yes

Notes

@JeffMboya JeffMboya self-assigned this Dec 5, 2025
@JeffMboya
Copy link
Contributor Author

1. Status Quo

Currently, Propeller operates on a "Train Once, Deploy Everywhere" model. The intelligence of the system is static and created entirely outside the Propeller runtime environment.

  • Training is External: the machine learning models are trained offline using Python scripts (e.g., train_model.py) on a centralized dataset.
  • Models are Hardcoded: The "intelligence" of the application is frozen into the code. The training script outputs hard numbers (coefficients and intercepts), which are manually pasted or generated into a Go function (model_gen.go).
    // The "learning" is finished before this code is ever compiled
    func score(input []float64) float64 {
        return 0.0... + input[0] * 1.99... // Hardcoded constants
    }
  • One-Way Flow: The Manager sends a task to the Proplet. The Proplet executes the Wasm, runs the hardcoded math on inputs, and returns a result. The Edge device acts solely as a calculator, not a learner.

2. The Problem

To achieve Federated Learning, where devices learn from local data without sharing it, the current architecture faces three blocking issues:

A. The Immutable Wasm Barrier

The current Wasm modules are stateless and immutable.

  • Current State: The score function inside the Wasm binary uses fixed constants. The Wasm memory does not persist updates to these constants between executions.
  • The Conflict: Federated Learning requires mutable state. The Wasm module needs to hold a set of weights, receive local data, mathematically adjust those weights (calculate gradients), and output the difference, not just a prediction.

B. The "Dumb" Manager

The Manager is currently designed as a Task Dispatcher, not a Model Aggregator.

  • Current State: The Manager schedules tasks (scheduler.go), sends start commands, and stores the returned results blindly into a database (tasksDB). It treats the result as a final output (e.g., "The temperature is 25°C").
  • The Conflict: In Federated Learning, the "result" from a worker is not an output, but a partial model update. The Manager needs new logic to recognize these updates, wait for a quorum of workers, perform mathematical aggregation (e.g., Federated Averaging), and issue a new global model back to the network.

C. The Disconnected Data Loop

There is no mechanism for the Edge device to utilize its own data for self-improvement.

  • Current State: Data flows from the input arguments $\rightarrow$ Wasm $\rightarrow$ Output. The system assumes the "truth" (the training data) lives on the server where the Python script runs.
  • The Conflict: In Federated Learning, the "truth" lives on the Edge. The Proplet needs access to local historical data or sensor readings to compute the error rate (Loss Function) of the current model, which drives the learning process. The current main.go simply takes inputs passed from the CLI/Manager, effectively ignoring the rich local data available on the device.

@JeffMboya JeffMboya changed the title PROP-96 - Add ML to WASM demo application PROP-96 - Add Federated ML to Propeller Dec 17, 2025
@JeffMboya JeffMboya force-pushed the PROP-96 branch 5 times, most recently from c331076 to 06be567 Compare December 18, 2025 10:14
Use pickle for model2wasm

Fix TinyGo entrypoint and m2cgen wrapping workflow

Add FL task mode and metadata fields

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Include FL mode and spec in start payload

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Add FL update envelope type for results

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Deduplicate FL helpers; update host and wazero

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Handle train results as FL envelope updates

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Coordinate federated rounds; validate and aggregate updates

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Coordinate federated rounds; validate and aggregate updates

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Coordinate federated rounds; validate and aggregate updates

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Coordinate federated rounds; validate and aggregate updates

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Make wazero StopApp idempotent like host

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Include numeric args in wazero WASI argv

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Update runtime and transport logic

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Fix CI

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Fix CI

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Fix CI

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Fix CI

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Fix CI

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Fix CI

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Fix CI

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>

Fix CI

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
@@ -0,0 +1,91 @@
//go:build wasm
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need these build tags and we aren't using them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They ensure the file only compiles for WASM targets (GOOS=wasip1 GOARCH=wasm)

TESTING.md Outdated
@@ -0,0 +1,89 @@
# Testing Guide
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this here?

README.md Outdated
- ☁️ **Serverless Applications**: Deploy FaaS applications leveraging Propeller's Wasm orchestration capabilities.
- 🧠 **Federated Machine Learning**: Train machine learning models across distributed edge devices without exposing raw data, perfect for privacy-sensitive applications.

## 🤖 Federated Learning
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be done on docs repo

package main

func score(input []float64) float64 {
return 0.0000000000000008881784197001252 + input[0] * 1.9999999999999996 + input[1] * 2.999999999999999
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These seem like magic numbers

@@ -0,0 +1,14 @@
package main
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have a documentation on how to run these two examples?


script_dir = os.path.dirname(os.path.abspath(__file__))

model_path = os.path.join(script_dir, "mymodel.pkl")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two files should be git ignored

  • examples/ml-wasm/mymodel.pkl
  • examples/ml-wasm/mymodel.wasm

@@ -0,0 +1,392 @@
package manager
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should keep the integration test. It validates the full FL pipeline and ensures components work together correctly.

var weights []float64
err = json.Unmarshal(decoded, &weights)
if err != nil {
t.Fatalf("Failed to unmarshal weights: %v", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use supermq strategy

@@ -0,0 +1,27 @@
package proplet
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be removed

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
@JeffMboya
Copy link
Contributor Author

image

@JeffMboya
Copy link
Contributor Author

image

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
@JeffMboya
Copy link
Contributor Author

image

@JeffMboya
Copy link
Contributor Author

image

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
@JeffMboya
Copy link
Contributor Author

image

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
@JeffMboya
Copy link
Contributor Author

JeffMboya commented Jan 16, 2026

First training round, model "version": 1 saved

image

@JeffMboya
Copy link
Contributor Author

JeffMboya commented Jan 16, 2026

second training round, global model updated to "version": 2. We now have three versions of the global model:

  • v0: The initial un-trained model (all zeros)
  • v1: The result of the first round of aggregation
  • v2: The result of the second round.

The weights in v2 (0.000045...) are different from v1 (0.000246...), proving that the training on the "Edge" (Proplets) is actually influencing the global model

image

Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
@JeffMboya
Copy link
Contributor Author

This can be closed in favour of #117

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Add ML to WASM demo application

2 participants